Fix flaky test_command_timeout_fail in SSH provider#65864
Open
Dev-iL wants to merge 1 commit into
Open
Conversation
Dev-iL
force-pushed
the
2604/deflake_ssh
branch
from
April 26, 2026 06:37
0789540 to
04c61d0
Compare
Dev-iL
force-pushed
the
2604/deflake_ssh
branch
2 times, most recently
from
June 9, 2026 11:39
8055f4d to
173a5e0
Compare
The test_command_timeout_fail test was flaky because it relied on real SSH connections and a 1ms timeout, causing paramiko's internal socket timeouts to fire before the select loop could catch them. Replaced real SSH connection with mocks for deterministic testing: - Mock paramiko.SSHClient.exec_command to return controlled channel objects - Mock select to simulate immediate timeout - Use spec on all mocks (Channel, ChannelFile, ChannelStdinFile, ChannelStderrFile) to catch attribute typos - Verify cmd_timeout is correctly threaded through to both exec_command and select calls - Assert cleanup side effects: stdin.close, shutdown_write, shutdown_read, channel.close, stdout.close, stderr.close
eladkal
force-pushed
the
2604/deflake_ssh
branch
from
July 24, 2026 06:59
173a5e0 to
73c1a10
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
TestSSHHook::test_command_timeout_failwas intermittently failing in CI.The original test opened a real SSH connection to
ssh_default, ransleep 1with a 1 ms (0.001) timeout, and expectedAirflowExceptionto be raised. Two independent timeout mechanisms were racing:paramiko's channel-level socket timeout.
exec_command(timeout=0.001)callsChannel.settimeout(0.001), which makes every subsequentrecv/sendon the channel raisesocket.timeoutafter 1 ms. This fires inside paramiko's own transport thread and during any blocking read the channel performs internally.Airflow's
select-loop timeout.exec_ssh_client_commandpassescmd_timeoutas the fourth argument toselect([channel], [], [], cmd_timeout). Whenselectreturns an empty list, the function setstimedout = Trueand eventually raisesAirflowException("SSH command timed out").With a 1 ms deadline both mechanisms fire at roughly the same instant. Depending on thread scheduling, CPU load, and kernel
selectgranularity:selectreturns first with an empty read list, Airflow's own timeout logic triggers, and the test catchesAirflowException.socket.timeoutfires first—inside the transport thread or during thestdin.close()/channel.shutdown_write()calls that happen before theselectloop is even reached. That surfaces asparamiko.ssh_exception.SSHExceptionorsocket.timeout, neither of which matches theAirflowExceptionthe test expects.Because the race depends on real wall-clock time, it is non-deterministic. A fast CI runner tips the odds one way; a loaded one tips them the other.
What the fix does
The fix removes the real SSH connection entirely and replaces it with mocks that deterministically exercise the timeout detection logic inside
exec_ssh_client_command.The key mock is on
select:Returning
([], [], [])simulatesselectreporting "nothing readable within the timeout window." The production code then follows its normal path:There is no second timeout mechanism in play.
paramiko.SSHClientis aMagicMock(spec=...), soexec_commandreturns instantly with mock objects.stdin.close()andchannel.shutdown_write()are no-ops on the mock, so they can never raise a socket-level exception. The only timeout that fires is Airflow's, which is exactly what the test is verifying.What the fix verifies beyond the original test
The original test had a single assertion: "an
AirflowExceptionis raised." The new test adds:match="SSH command timed out"onpytest.raisesfake_selectassertstimeout == 0.001cmd_timeoutnot threaded through toselectexec_command.assert_called_once_with(command=..., timeout=0.001, ...)cmd_timeoutnot passed to paramikomock_stdin.close.assert_called_once()mock_channel.shutdown_write.assert_called_once()mock_channel.shutdown_read.assert_called_once()mock_channel.close.assert_called_once()mock_stdout.close.assert_called_once()mock_stderr.close.assert_called_once()All mocks use
spec=against the real paramiko types (paramiko.Channel,paramiko.ChannelFile,paramiko.ChannelStdinFile,paramiko.ChannelStderrFile,paramiko.SSHClient), so accessing a misspelled attribute on any mock will raiseAttributeErrorimmediately rather than silently returning a newMagicMock.What the fix does NOT do
It does not test the data-reading path (stdout/stderr aggregation), the graceful-exit path (command finishes before timeout), or the
channel.close()race condition handler (lines 492-498). Those are separate behaviors covered by other tests (test_command_timeout_success,test_command_timeout_not_set, and the broaderexec_ssh_client_commandintegration tests). This test is scoped to one thing: the timeout detection and cleanup path.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Opus 4.6 following the guidelines
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.